Skip to content

datasets: add 149 contrastive Turkish prompt-injection cases#46

Open
3nesdeniz wants to merge 3 commits into
GenAI-Security-Project:mainfrom
3nesdeniz:agent/add-turkish-conversation-pair-suite
Open

datasets: add 149 contrastive Turkish prompt-injection cases#46
3nesdeniz wants to merge 3 commits into
GenAI-Security-Project:mainfrom
3nesdeniz:agent/add-turkish-conversation-pair-suite

Conversation

@3nesdeniz

@3nesdeniz 3nesdeniz commented Jul 18, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

  • adds 149 Turkish-first, standalone prompt-injection and data-extraction regression cases
  • keeps each adversarial prompt with a vocabulary- and context-matched benign control so evaluators can measure both attack containment and false-positive behavior
  • includes a strict extension schema, deterministic builder, SHA-256 manifest, pinned provenance, and an executable validator
  • preserves the source release text byte-for-byte; one malformed reversed-payload pair (pair_0140 / TC-0440) is transparently excluded rather than silently corrected

Source and licensing

The cases are adapted from the Turkish Conversation Prompt-Injection Dataset v1.0.2, also published on Hugging Face with DOI 10.5281/zenodo.21379389.

Source attack/control text remains under CC BY 4.0. New mappings, expectations, metadata, scripts, and collection arrangement are contributed under this repository's CC BY-SA 4.0 license. Attribution, source revision, release tag, per-file hashes, and adaptation changes are recorded in the suite README, manifest, and case provenance.

Coverage

  • 149 paired cases across 10 attack families (nine families × 15; obfuscation/code-switching × 14 after the documented exclusion)
  • 85 data-extraction cases and 64 prompt-injection-control cases
  • Turkish-first coverage: 147 Turkish-primary and 2 English-primary attacks; both English-primary cases retain Turkish controls
  • DSGAI mappings across DSGAI01, 02, 04, 05, 06, 09, 11, 13, 15, and 17
  • all systems, credentials, tools, and protected records are synthetic or mocked

Validation

  • python3 build_cases.py --source <pinned-v1.0.2-checkout>: 149 cases; two consecutive builds produced identical hashes for all 149 cases plus the manifest
  • python3 validate.py --source <pinned-v1.0.2-checkout>: 149/149 schema, provenance, source-text, manifest, mapping, safety, and integrity checks passed
  • source dataset validator: 750/750 rows passed; published JSONL and Parquet checksums verified
  • upstream test suite: 27 passed, 1 skipped
  • exact attack/control collisions: 0
  • exact overlap with existing repository prompts: 0
  • sensitive-data, live-secret, and named-vendor findings: 0
  • maximum unrelated lexical similarity remained below the 0.92 failure threshold

Both commits are SSH-signed and verified by GitHub. The contribution is ready for maintainer review.

@3nesdeniz
3nesdeniz marked this pull request as ready for review July 18, 2026 16:46
Copilot AI review requested due to automatic review settings July 18, 2026 16:46
Copilot AI reviewed Jul 18, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR expands the prompt-injection/data-extraction regression dataset by adding a new Turkish-first contrastive suite derived from the Turkish Conversation Prompt-Injection Dataset v1.0.2, pairing each adversarial prompt with a vocabulary-matched benign control and including provenance + integrity metadata for reproducible evaluation.

Changes:

  • Added new standalone JSON test cases for the tr_turkish_conversation_prompt_injection_pairs suite, including contrastive controls, provenance, and SHA-256 integrity fields.
  • Updated the top-level dataset README to reference the canonical schema and enumerate current suites.

Reviewed changes

Copilot reviewed 155 out of 155 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0449.json Adds a new contrastive obfuscation/code-switching control testcase (TC-0449).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0447.json Adds a new contrastive obfuscation/code-switching control testcase (TC-0447).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0444.json Adds a new data-extraction testcase with matched benign control (TC-0444).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0435.json Adds a new memory-context poisoning control testcase (TC-0435).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0432.json Adds a new memory-context poisoning control testcase (TC-0432).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0428.json Adds a new memory-context poisoning control testcase (TC-0428).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0426.json Adds a new memory-context poisoning control testcase (TC-0426).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0425.json Adds a new memory-context poisoning control testcase (TC-0425).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0424.json Adds a new memory-context poisoning control testcase (TC-0424).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0423.json Adds a new memory-context poisoning data-extraction testcase (TC-0423).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0390.json Adds a new tool-action abuse control testcase (TC-0390).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0389.json Adds a new tool-action abuse control testcase (TC-0389).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0388.json Adds a new tool-action abuse control testcase (TC-0388).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0387.json Adds a new tool-action abuse control testcase (TC-0387).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0384.json Adds a new tool-action abuse control testcase (TC-0384).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0380.json Adds a new tool-action abuse control testcase (TC-0380).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0376.json Adds a new tool-action abuse control testcase (TC-0376).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0375.json Adds a new sensitive-data exfiltration testcase (TC-0375).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0374.json Adds a new sensitive-data exfiltration testcase (TC-0374).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0373.json Adds a new sensitive-data exfiltration testcase (TC-0373).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0370.json Adds a new sensitive-data exfiltration testcase (TC-0370).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0369.json Adds a new sensitive-data exfiltration testcase (TC-0369).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0367.json Adds a new sensitive-data exfiltration testcase (TC-0367).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0364.json Adds a new sensitive-data exfiltration testcase (TC-0364).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0363.json Adds a new sensitive-data exfiltration testcase (TC-0363).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0362.json Adds a new sensitive-data exfiltration testcase (TC-0362).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0361.json Adds a new sensitive-data exfiltration testcase (TC-0361).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0356.json Adds a new authority-claim bypass control testcase (TC-0356).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0355.json Adds a new authority-claim bypass data-extraction testcase (TC-0355).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0354.json Adds a new authority-claim bypass data-extraction testcase (TC-0354).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0350.json Adds a new authority-claim bypass control testcase (TC-0350).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0349.json Adds a new authority-claim bypass data-extraction testcase (TC-0349).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0347.json Adds a new authority-claim bypass data-extraction testcase (TC-0347).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0345.json Adds a new roleplay jailbreak data-extraction testcase (TC-0345).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0344.json Adds a new roleplay jailbreak control testcase (TC-0344).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0343.json Adds a new roleplay jailbreak data-extraction testcase (TC-0343).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0342.json Adds a new roleplay jailbreak control testcase (TC-0342).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0340.json Adds a new roleplay jailbreak control testcase (TC-0340).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0336.json Adds a new roleplay jailbreak data-extraction testcase (TC-0336).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0335.json Adds a new roleplay jailbreak data-extraction testcase (TC-0335).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0334.json Adds a new roleplay jailbreak control testcase (TC-0334).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0331.json Adds a new roleplay jailbreak control testcase (TC-0331).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0330.json Adds a new system-prompt extraction testcase (TC-0330).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0329.json Adds a new system-prompt extraction testcase (TC-0329).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0328.json Adds a new system-prompt extraction testcase (TC-0328).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0327.json Adds a new system-prompt extraction testcase (TC-0327).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0325.json Adds a new system-prompt extraction testcase (TC-0325).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0324.json Adds a new system-prompt extraction testcase (TC-0324).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0323.json Adds a new system-prompt extraction testcase (TC-0323).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0322.json Adds a new system-prompt extraction testcase (TC-0322).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0321.json Adds a new system-prompt extraction testcase (TC-0321).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0320.json Adds a new system-prompt extraction testcase (TC-0320).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0319.json Adds a new system-prompt extraction testcase (TC-0319).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0318.json Adds a new system-prompt extraction testcase (TC-0318).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0317.json Adds a new system-prompt extraction testcase (TC-0317).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0316.json Adds a new system-prompt extraction testcase (TC-0316).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0315.json Adds a new direct-instruction override control testcase (TC-0315).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0314.json Adds a new direct-instruction override control testcase (TC-0314).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0311.json Adds a new direct-instruction override control testcase (TC-0311).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0309.json Adds a new direct-instruction override control testcase (TC-0309).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0308.json Adds a new direct-instruction override control testcase (TC-0308).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0307.json Adds a new direct-instruction override control testcase (TC-0307).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0306.json Adds a new direct-instruction override control testcase (TC-0306).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0305.json Adds a new direct-instruction override control testcase (TC-0305).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0304.json Adds a new direct-instruction override control testcase (TC-0304).
datasets/promptinj_dataextraction_testcases/tr_turkish_conversation_prompt_injection_pairs/cases/TC-0303.json Adds a new direct-instruction override control testcase (TC-0303).
datasets/promptinj_dataextraction_testcases/README.md Documents the canonical schema and lists available suites.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread datasets/promptinj_dataextraction_testcases/README.md
@3nesdeniz

3nesdeniz commented Jul 20, 2026

Copy link
Copy Markdown
Contributor Author

The branch is now synchronized with current main, the schema/category guidance thread is addressed and resolved, and the hosted Dependency Review check is green. Fresh local validation on head 1f246a2: 149/149 cases passed the canonical and extension schemas plus pinned source/provenance/integrity checks; the deterministic rebuild produced no diff; all 6 data-validation unit tests and Ruff checks passed. Ready for maintainer review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@3nesdeniz